The adipose cell-size distribution is a quantitative characterization of adipose tissue morphology. At a population level, the adipose cell-size distribution is insulin-sensitivity dependent, and the observed correlation between obesity and insulin resistance is believed to play a key role in the metabolic syndrome. Changes in fat mass can be induced by altered energy intake or even diet composition. These macroscopic changes must manifest themselves as dynamic adipose cell-size distribution alterations at the microscopic level. The dynamic relationship between these two independent measurements of body fat is unknown. In this study, we investigate adipose tissue dynamics in response to various isocaloric diet compositions, comparing gender- and insulin sensitivity-dependent differences. A body composition model is used to predict fat mass changes in response to changes in diet composition for 28 individuals, separated into 4 subgroups according to gender and insulin sensitivity/resistance. Adipose cell-size distribution changes in each individual are simulated with a dynamic model and parameters corresponding to lipid turnover and cell growth rates are determined for each subgroup to match the relative change of fat mass for each diet composition, respectively. We find that adipose cell-size dynamics are associated with different modulations dependent on gender and insulin resistance. Larger turnover and growth/shrinkage rates in insulin resistant individuals suggest they may be more sensitive to changes in energy intake and diet composition than insulin sensitive subjects. The different cell-size distribution changes of adipose cells of various sizes in different subject groups further suggest distinct modulations of adipose cell dynamics.